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FUTURE-ORIENTED  EXPERIMENTAL  ARMY  ENLISTED  PERSONNEL  SELECTION 
AND  CLASSIFICATION  PROJECT  (SELECT21)  SUMMARY  REPORT 

EXECUTIVE  SUMMARY  


Research  Requirement: 

The  Select21  project  was  undertaken  to  help  the  U.S.  Army  ensure  that  it  acquires 
Soldiers  with  the  knowledges,  skills,  and  other  attributes  (KSAs)  needed  for  performing  the  types 
of  tasks  envisioned  in  a  transformed  Army.  This  transformation  will  involve  development  and 
fielding  of  Future  Combat  Systems  (FCSs)  to  achieve  dominance  through  a  force  that  is 
responsive,  deployable,  agile,  versatile,  lethal,  and  fully  survivable  and  sustainable  under  all 
anticipated  combat  conditions.  However,  Army  leadership  recognizes  first  and  foremost  the 
importance  of  its  people  -  Soldiers  -  to  the  effectiveness  of  transformation.  In  this  context,  the 
ultimate  objectives  of  the  project  were  to  (a)  develop  and  validate  measures  of  critical  KSAs 
needed  for  successful  execution  of  future  force  missions  and  (b)  propose  use  of  these  measures  as  a 
foundation  for  an  entry-level  selection  and  classification  system  adapted  to  the  demands  of  the  21st 
century. 

Procedure: 

Early  in  the  Select21  project,  we  conducted  a  future-oriented  job  analysis  (Sager,  Russell, 
R.C.  Campbell,  &  Ford,  2005)  to  support  the  development  of  criterion  measures  and  experimental 
selection  and  classification  predictor  measures  (Knapp,  Sager,  &  Tremble,  2005).  We  then  used  a 
concurrent  validation  design  in  which  we  administered  both  the  predictor  and  criterion  measures  to 
first-term  enlisted  Soldiers  who  had  been  in  a  unit  for  at  least  12  months. 

The  criterion  measures  included  (a)  job  knowledge  tests,  (b)  a  criterion  situational 
judgment  test  (CSJT),  (c)  performance  ratings  (covering  current  performance  and  anticipated 
performance  under  explicitly  defined  future  conditions)  collected  from  supervisors  and  peers,  and 
(d)  surveys  of  current  job  attitudes  (the  Army  Life  Survey;  ALS)  and  expected  attitudes  under 
defined  future  conditions.  All  Soldiers  completed  versions  of  these  measures  suitable  for  first-term 
Soldiers  regardless  of  military  occupational  specialty  (MOS).  We  administered  job  specific 
criterion  measures  to  Infantrymen  (1  IB)  and  Signal  Support  Systems  Specialists  (25U),  but  the 
25U  sample  was  too  small  to  support  planned  classification  efficiency  analyses.  Therefore,  data 
analysis  work  focused  primarily  on  the  extent  to  which  each  of  the  experimental  measures  was 
related  to  Army-wide  performance.  These  analyses  included  estimation  of  incremental  validity 
beyond  the  predictive  power  of  scores  from  the  Armed  Services  Vocational  Aptitude  Battery 
(ASVAB). 

The  experimental  predictors  administered  in  the  concurrent  validation  included  (a)  two 
temperament  measures  (Rational  Biodata  Inventory  [RBI]  and  Work  Suitability  Inventory  [WSI]), 
(b)  a  predictor  situational  judgment  test  (PSJT),  and  (c)  two  psychomotor  tests  (Target  Shoot  and 


Target  Tracking).  There  were  also  two  measures  based  on  person-environment  fit  models,  the 
Work  Values  Inventory  (WVI)  and  the  Work  Preferences  Survey  (WPS).  The  WPS  is  a  measure  of 
interests  and  the  WVI  asks  respondents  about  the  characteristics  they  value  in  a  work  environment 
(e.g.,  autonomy).  Some  measures  developed  in  Select21  and  described  in  Knapp  et  al.  (2005)  were 
not  included  in  the  concurrent  validation  because  they  were  not  suitable  for  administration  to 
experienced  Soldiers  (e.g.,  the  Pre-Service  Expectations  Survey). 

Findings: 

Our  intent  was  to  develop  predictors  that  supplement  the  ASVAB  for  the  prediction  of 
performance  and  attitudinal  criteria.  We  constructed  five  composite  performance  scores  (based 
on  a  confirmatory  factor  analysis  modeling  exercise)  and  five  attitudinal  scores  to  use  in  the 
validation  analyses.  The  five  performance  criteria  were  (a)  General  Technical  Proficiency,  (b) 
Achievement  and  Effort,  (c)  Physical  Fitness,  (d)  Teamwork,  and  (e)  Future  Expected 
Performance.  The  five  attitudinal  scores  were  (a)  Satisfaction  with  the  Army,  (b)  Perceived 
Army  Fit,  (c)  Attrition  Cognitions,  (d)  Career  Intentions,  and  (e)  Future  Army  Affect. 

Consistent  with  prior  research,  scores  on  the  ASVAB  were  shown  to  be  good  predictors 
of  can-do  performance  criteria  (e.g.,  General  Technical  Proficiency)  and  to  have  lower  validity 
for  predicting  will-do  (e.g.,  Physical  Fitness,  Teamwork)  and  attitudinal  criteria.  ASVAB  scores 
yielded  significant  correlations  with  Future  Expected  Performance.  This  is  a  new  finding  because 
the  question  of  future  performance  (and  attitudes)  has  never  before  been  posed  for  entry-level 
Soldiers.  ASVAB  scores  yielded  small  but  significant  negative  correlations  with  Attrition 
Cognitions.  Consistent  with  prior  research,  Soldiers  with  higher  cognitive  ability  were  less  likely 
to  think  about  breaking  their  enlistment  contracts,  although  they  were  more  likely  to  be  planning 
to  leave  at  the  end  of  their  enlistment  term. 


Scores  on  the  ASVAB  were  in  fact  such  good  predictors  of  can-do  performance  criteria 
(e.g.,  General  Technical  Proficiency)  that  they  left  little  room  for  improvement  (“incremental 
validity”)  with  the  addition  of  other  tests.  In  constrast,  many  of  the  Select21  predictors  showed 
notable  levels  of  incremental  validity  over  the  ASVAB  when  predicting  Achievement  and  Effort, 
Physical  Fitness,  and  Teamwork  performance.  In  addition,  substantial  levels  of  incremental 
validity  were  found  for  the  RBI,  WVI,  and  WPS  for  predicting  the  attitudinal  criteria,  with 
somewhat  lower  levels  of  validity  for  the  WSI  and  PSJT.  Such  findings  reinforce  the  notion  that 
when  judging  the  efficacy  of  predictors  for  incrementing  the  validity  of  the  ASVAB,  it  is 
important  to  account  for  the  multi-dimensional  nature  of  job  performance  and  job-related 
attitudes. 

We  performed  subgroup  analyses  using  type  of  MOS  as  the  subgrouping  variable  to  get 
an  idea  of  the  potential  for  the  experimental  predictors  to  improve  classification  efficiency. 
Soldiers  were  sorted  into  four  MOS  clusters  for  these  analyses,  which  did  suggest  that  some  of 
the  predictors  have  potential  utility  for  classification. 


VI 


Utilization  and  Dissemination  of  Findings: 

Many  of  the  new  Select21  predictors  are  self-report  indicators  in  which  scores  may  be 
affected  by  experience  in  the  Army  and  response  distortion  (whether  intentional  or  not)  in  an 
operational  setting.  Therefore,  it  is  particularly  important  to  evaluate  them  in  a  longitudinal 
validation  in  which  the  predictors  are  administered  to  Army  applicants  or  new  recruits.  A  follow- 
on  5-year  research  program  known  as  “Army  Class”  has  been  initiated  to  collect  such  data. 
Moreover,  Army  Class  is  designed  to  gather  more  MOS-specific  data.  This  will  allow  a  more 
definitive  assessment  of  the  classification  potential  of  the  experimental  predictors.  Army  Class 
includes  a  concurrent  validation  as  well  as  a  longitudinal  validation,  so  it  will  significantly  build 
upon  the  foundation  provided  by  Select21  for  implementation  of  Select21  enlistment  tests  for 
both  selection  and  classification  decision-making. 

A  key  ingredient  to  successful  implementation  of  research  findings  is  a  strong  interface 
between  researchers  and  policymakers.  This  report  describes  steps  taken  over  the  course  of  the 
Select21  project  and  future  activities  needed  to  promote  implementation  of  the  Select21  selection 
and  classification  tests. 
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Introduction 

The  U.S.  Army  has  undertaken  far-reaching  changes  to  transform  the  current  force  into  one 
that  is  more  responsive,  deployable,  agile,  versatile,  and  lethal  while  being  fully  survivable  and 
sustainable  under  all  conditions.  New  Predictors  for  Selecting  and  Assigning  Future  Force  Soldiers 
(Select21)  is  part  of  an  Army  research  program  focused  on  the  personnel  system  by  which  the 
Army  selects  entry-level  Soldiers  and  assigns  these  Soldiers  to  jobs.  This  research  program  is 
designed  to  help  ensure  that  through  this  system,  the  Army  will  acquire  new  Soldiers  with  the 
knowledges,  skills,  and  other  attributes  (KSAs)  needed  for  performing  the  types  of  tasks  emerging 
as  part  of  the  transformation.  More  specifically,  the  objectives  are  to  develop  measures  of  these 
KSAs  and  to  evaluate  their  potential  for  integration  into  the  Army’s  personnel  acquisition  system. 

It  bears  noting  that  concurrent  with  the  Select2 1  effort,  the  Department  of  Defense  (DoD) 
sponsored  a  panel  of  measurement  experts  to  review  the  current  selection  and  classification  test 
battery — the  Armed  Services  Vocational  Aptitude  Battery  (ASVAB) —  and  to  recommend 
changes.  As  it  turns  out,  the  Select21  research  approach  is  quite  consistent  with  this  panel’s 
recommendations  and  may  help  to  further  progress  on  those  recommendations  (Drasgow, 
Embretson,  Kyllonen,  &  Schmitt,  2006). 

SelectU  Research  Plan 

Figure  1  shows  a  schematic  overview  of  the  Select21  project.  It  began  with  job  analysis 
work  to  forecast  future  Army  job  requirements.  This  information  served  as  the  basis  for 
development  of  criterion  measures  and  a  battery  of  experimental  selection  and  classification 
tests.  To  obtain  evidence  of  how  well  these  experimental  pre-enlistment  tests  would  work,  we 
administered  them  to  a  sample  of  Soldiers  and  then  measured  how  well  these  same  Soldiers  fit 
with  their  jobs  using  measures  of  job  performance  and  attitudes.  Select21  used  a  “concurrent 
validation”  design  in  which  we  administered  the  experimental  tests  to  a  sample  of  Soldiers  with 
12  to  36  months  time  in  service  and,  at  the  same  time,  administered  the  criterion  measures. 
Although  not  depicted  in  Figure  1 ,  we  were  able  to  conduct  attrition-related  research  using  data 
from  Soldiers  who  participated  in  early  stages  of  the  research  in  which  they  took  draft  versions 
of  the  experimental  predictor  tests  as  new  recruits. 

The  original  research  plan  included  an  Army-wide  sample  that  comprised  a  mix  of 
military  occupational  specialties  (MOS),  plus  subsamples  of  Soldiers  in  six  selected  MOS.  The 
plan  called  for  collecting  MOS-specific  criterion  data  from  Soldiers  in  the  target  MOS  and  using 
the  information  to  evaluate  the  extent  to  which  the  experimental  predictors  would  be  useful  for 
supporting  personnel  classification  decisions.  Deployment  activity  associated  with  the  Global 
War  on  Terror,  however,  made  it  difficult  for  units  to  provide  Soldiers  for  testing.  Therefore,  we 
scaled  back  to  target  two  MOS  (Infantry  [1  IB]  and  Signal  Support  Systems  Specialty  [25U]). 
Even  with  this  reduction,  we  were  still  not  able  to  collect  sufficient  data  from  the  25U  MOS  to 
support  the  classification  research  planned.  This  work  is  thus  being  extended  under  a  follow-on 
research  effort,  Investigations  into  Army  Enlisted  Classification  Systems  (Army  Class).  In 


1 


addition  to  collecting  more  concurrent  validation  data,  the  Army  Class  research  program  will 
take  the  next  step  towards  implementation  and  conduct  a  longitudinal  validation  that  will  involve 
administering  the  most  promising  experimental  predictors  to  applicants  (or  new  recruits)  and 
tracking  them  over  time  to  collect  criterion  data.  This  design  will  be  the  ultimate  test  of  the  new 
measures  prior  to  final  implementation  decisions. 


Jan  2002  Jan  2003  Jan  2004  Jan  2005 

Figure  1.  Schematic  of  the  SelectU  project  plan. 


Purpose  and  Organization  of  Report 

The  purpose  of  this  report  is  to  describe  the  objectives,  approach,  and  findings  of  the 
Select21  research  project  in  a  manner  suitable  for  a  policymaking  audience.  It  also  offers  insights 
into  issues  related  to  implementation  of  the  research  findings.  Technical  details  of  the  research 
are  provided  in  several  companion  reports,  most  notably  Knapp,  Sager,  and  Tremble  (2005)  and 
Knapp  and  Tremble  (2006).  For  a  complete  listing  of  related  documents,  see  the  bibliography  at 
the  end  of  this  report. 

The  remainder  of  this  report  is  organized  into  six  sections.  The  first  four  sections  provide 
background  information  related  to  the  following  areas: 

•  Defining  future  entry-level  Soldier  requirements. 

•  Desired  outcomes  of  an  effective  selection  and  classification  system. 

•  Considerations  that  go  into  evaluating  potential  new  pre-enlistment  tests. 

•  Descriptions  of  the  experimental  tests  developed  in  Select2 1 . 

We  then  move  to  describing  what  we  know  about  the  effectiveness  of  the  ASVAB  and  the 
experimental  Select21  tests  and  what  other  research  questions  we  are  asking  in  the  follow-on 
Army  Class  project.  The  report  concludes  with  a  discussion  related  to  maximizing  the  successful 
integration  of  research  findings  into  operational  programs. 
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The  First  Step:  Identifying  Future  Requirements 

At  the  beginning  of  the  Select21  research  effort,  we  took  considerable  care  to  identify 
future  requirements  for  first-term  Soldiers  (Sager,  Russell,  R.C.  Campbell,  &  Ford,  2005).  This 
activity  carried  forth  work  previously  done  in  support  of  a  comparable  research  program  focused 
on  the  enlisted  personnel  promotion  system  (NC021)  (Ford,  R.C.  Campbell,  J.P.  Campbell, 
Knapp,  &  Walker,  2000).  While  much  had  been  written  prior  to  the  NC021  project  about 
directions  for  change,  particularly  as  it  regards  weapons  systems,  little  of  this  information 
discussed  implications  for  the  jobs  of  individual  Soldiers.  In  short,  we  combined  information 
from  Army  planning  documents  with  information  about  the  current  force  structure  and  job 
requirements  and  used  this  as  a  basis  for  projecting  details  about  the  Army’s  future  force.  Our 
target  was  15-20  years  into  the  future.  We  were  aided  by  Army  subject  matter  experts  (SMEs) 
throughout  this  process  and  created  the  following  products  as  a  result  of  this  effort: 

•  A  set  of  1 6  entry-level  job  clusters 

•  Future  conditions  for  entry-level  Soldiers 

•  Dimensions  of  entry-level  Soldier  job  performance 

•  Entry-level  Soldier  job  tasks 

•  Pre-enlistment  KSA  requirements 

The  job  clusters  were  used  as  a  basis  for  sampling  MOS  to  include  in  the  Select21 
research.  The  future  conditions,  job  performance  dimensions,  and  job  task  information  were  used 
as  a  basis  for  (a)  identifying  the  KSA  requirements  and  (b)  development  of  the  criterion 
measures. 


Defining  Desired  Selection  and  Classification  Outcomes 

In  research  of  this  nature,  it  is  important  to  clearly  define  the  criteria  of  interest.  That  is, 
how  do  we  determine  if  selection  and  classification  decisions  are  being  made  effectively? 
Currently,  the  primary  tool  for  selection  and  classification  for  all  the  U.S.  Armed  Services 
(including  the  Coast  Guard)  is  the  ASVAB.  The  original  goal  of  the  ASVAB  was  to  predict  how 
well  individuals  would  perform  in  technical  training,  so  technical  training  performance  has 
historically  been  the  criterion  of  interest  in  validation  research  such  as  that  conducted  in 
Select21 .  In  today’s  Army  and  certainly  in  tomorrow’s  Army,  however,  the  criteria  of  interest 
are  much  broader  in  nature. 

Although  training  performance  is  important,  force  readiness  has  more  to  do  with 
performance  in  units  in  the  field  and  with  having  Soldiers  who  stay  in  the  Army  than  it  does  with 
schoolhouse  performance.  Therefore,  we  developed  experimental  pre-enlistment  tests  to  predict 
the  following  criteria: 

•  Job  performance  (i.e.,  performance  in  units) 

•  Job-related  attitudes  (e.g.,  job  satisfaction)  related  to  attrition  and  retention  behavior 

•  Attrition 
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Moreover,  the  Select21  research  effort  included  the  determination  of  future  job  requirements  and 
conditions  expected  to  characterize  a  transformed  Army,  and  we  used  this  information  to  develop 
variations  of  the  criterion  measures  to  try  to  capture  the  impact  of  those  conditions  on  Soldiers’ 
performance  and  attitudes.  The  specific  indices  used  to  evaluate  the  experimental  predictor  tests 
are  briefly  described  below. 

Performance  Ratings 

We  developed  performance  rating  scales  to  be  completed  by  Soldiers’  supervisors  and 
peers.  One  set  of  rating  scales  was  suitable  for  first-term  Soldiers  in  any  MOS  and  included 
sections  on  technical  or  “can-do”  aspects  of  performance  (e.g.,  common  and  MOS-specific  task 
performance)  and  softer  but  no  less  important  motivational  or  “will  do”  aspects  of  performance 
(e.g.,  adaptability,  effort  and  initiative,  supporting  peers).  A  second  type  of  rating  scale  was 
developed  for  selected  target  MOS  to  obtain  ratings  of  performance  areas  specific  to  the  MOS. 
Contents  for  the  rating  scales  were  based  on  the  future-oriented  job  analysis  conducted  prior  to 
development  of  all  the  Select21  measures.  Figure  2  shows  one  of  the  Army-wide  rating  scales. 


Demonstrates  Professionalism  and  Personal  Discipline  on  the  Job 

The  extent  to  which  the  Soldier  exhibits  selfless  service  orientation,  exhibits  integrity  and  discipline  on 
the  job,  and  follows  instructions,  rules  and  regulations 

-  Fails  to  adhere  to  Army  standards 
and  values 

-  Generally  adheres  to  Army 
standards  and  values 

-  Consistently  adheres  to  Army 
standards  and  values 

-  Is  disrespectful  toward  superiors; 
may  ignore  or  refuse  to  follow 
orders 

-  Follows  direct  orders  with  little  or 
no  complaint  and  is  usually 
respectful  to  superiors 

Follows  orders  willingly;  is  always 
respectful  to  superiors 

-  Usually  displays  poor  military 
bearing 

-  Usually  maintains  good  military 
bearing 

-  Consistently  maintains  excellent 
military  bearing 

-  Does  not  put  in  effort  to  meet 
team  goals,  even  when  given 
encouragement 

-  Puts  in  effort  to  achieve  team 
goals,  but  may  need  some 
encouragement 

-  Willingly  puts  in  required  effort  to 
ensure  team  goals  are  achieved 

-  T akes  part  in  prohibited  activities 

-  Usually  resists  temptation  to  take 
part  in  prohibited  activities,  but 
might  succumb  to  peer  pressure 

Resists  even  strong  peer  pressure 
to  take  part  in  prohibited  activities 

-  Exhibits  little  or  no  self-control  or 
discipline  on  the  job 

-  Generally  exhibits  self-control  and 
discipline  on  the  job 

-  Exhibits  strong  self-control  and 
discipline  on  the  job 

Below  Expectations 

Meets  Expectations 

Exceeds  Expectations 

1  2 

3  4  5 

6  7 

Figure  2.  Sample  Army-wide  current  performance  rating  scale. 


After  completing  ratings  describing  Soldiers’  current  performance,  raters  in  the  Select21 
research  were  given  a  briefing  about  changes  expected  to  take  place  in  the  Army  over  the  next 
1 5-20  years.  This  briefing  characterized  future  conditions  in  terms  of  five  major  areas  of  change 
relative  to  the  current  environment: 
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•  Increased  individual  pace  and  intensity 

•  Need  for  more  self-directed  learning 

•  Greater  requirement  for  disciplined  initiative  (functioning  with  less  supervisor  and 
peer  support) 

•  Increased  amounts  of  frequently  updated  information 

•  Increased  survivability 

The  briefing  also  included  information  about  the  weapons  and  Soldier  support  systems  expected 
to  impact  future  conditions  (e.g.,  future  combat  systems). 

After  the  briefing,  raters  completed  a  set  of  “Expected  Future  Performance”  rating  scales 
that  required  them  to  predict  how  well  each  Soldier  they  were  rating  would  perform  under  the 
future  conditions.  A  comparable  set  of  future-oriented  rating  scales  also  was  developed  for  each 
target  MOS.  Figure  3  shows  an  example  rating  dimension  from  the  Army-wide  scales. 


Future  Condition:  Increased  Individual  Pace  and  Intensity 

Future  operations  will  likely  involve  new  aspects  of  physical,  psychomotor,  and  mental  skills.  Future  conflicts 
are  expected  to  involve  intense  and  sustained  operations  that  will  require  physical  and  mental  stamina  to 
conduct  high  paced  operation  over  long  periods.  Individuals  must  be  capable  of  cycling  between  periods  of 
work  and  rest  instantaneously  and  at  unpredictable  intervals.  Mental  sharpness  will  be  important  and 
individuals  will  be  required  to  process,  sort,  and  prioritize  digital  information  and  data  flow  without  becoming 
overwhelmed,  even  when  fatigued  or  stressed.  Soldiers  must  be  able  to  recognize  and  respond  to  mental 
cues  and  images  (such  as  icons  and  graphics)  rather  than  real-life  visual  or  sound  stimuli. 

Soldiers  will  undergo  rapid  transitions  in  mission  types  and  operational  environments.  Situational  conditions, 
such  as  rules  of  engagement,  hostile  forces,  threat  intent,  and  force  mission  can  change  daily.  Adaptability  will 
be  key.  There  will  be  less  time  for  transition  and  experience  learning  as  Soldiers  can  go  from  a  peacetime, 
home  centered,  CONUS  environment  to  full  combat  activities  in  a  matter  of  a  few  days. 


Individual  Pace  and  Intensity 

How  effectively  would  you  expect  the  Soldier  to  meet  these  future  requirements? 

Not  likely  to  meet  the  Soldier 
demands  described. 

Likely  to  be  generally  successful, 
but  will  struggle  to  meet  the 

Soldier  demands  described. 

Likely  to  successfully  meet  or 
exceed  the  Soldier  demands 
described. 

LOW 

MODERATE 

HIGH 

1  2 

3  4  5 

6  7 

Figure  3.  Sample  Army-wide  future  expected  performance  rating  scale. 
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Job  Knowledge 

We  developed  an  Army-wide  job  knowledge  test  as  well  as  tests  specific  to  the  target 
MOS.  The  Army-wide  test  primarily  captured  knowledge  of  procedures  associated  with  common 
tasks  (e.g.,  evaluate  a  casualty;  navigate  using  a  compass,  a  map,  and  overlays).  The  test  items 
are  a  blend  of  multiple-choice,  matching,  ranking,  and  drag-and-drop  formats  and  make  liberal 
use  of  visual  aids  (e.g.,  graphics  and  figures).  As  with  the  other  performance  criterion  measures, 
test  content  is  geared  to  first-term  Soldiers.  Sample  items  are  shown  in  Figure  4. 

Sample:  Which  MOPP  level  is  shown  in  the  figure  below? 


A.  MOPP  1 

B.  MOPP  2 

C.  MOPP  3 

D.  MOPP  4 


Sample:  An  enemy  armor  platoon  (reduced  strength)  has  established  a  strong  point  defense  along  an  avenue  of 
approach.  The  enemy  commander  has  arrayed  his  forces  to  maximize  his  current  assets  to  delay  or  stop  an  opposing 
light  infantry  company.  What  is  the  HVT? 


A.  AT  missile 

B.  T-72  tank 

C.  SA-missile  squad 

D.  Mortar  platoon 

Figure  4.  Sample  job  knowledge  test  items . 


6 


Judgment 


In  addition  to  measuring  job  knowledge,  it  is  important  to  determine  how  well  Soldiers 
judge  what  should  be  done  in  problem  situations,  such  as  working  with  uncooperative  peers  to 
accomplish  a  task  or  determining  when  to  handle  a  problem  alone  versus  consulting  a  supervisor. 
The  “Criterion  Situational  Judgment  Test”  contains  problem  scenarios  and  possible  actions  that 
had  been  generated  by  first-term  Soldiers  and  their  supervisors.  A  sample  item  is  shown  in 
Figure  5.  The  scoring  key  was  developed  using  judgments  provided  by  senior  NCOs. 


Instructions:  Your  task  is  to  rate  the  effectiveness  of  each  of  these  actions  using  a  1-7  rating  scale,  where  7  is 
highly  effective  and  1  is  completely  ineffective.  Imagine  that  you  are  in  the  situation;  use  the  scale  to  indicate 
how  effective  or  ineffective  you  believe  each  action  to  be. 

Sample  Question:  You  are  a  member  of  second  squad,  First  Platoon.  You  have  noticed  that  a 
member  of  third  squad  has  begun  to  let  job  performance  and  military  appearance  slip.  Once  very 
punctual,  this  Soldier  has  been  late  to  two  formations  in  the  last  week.  What  should  you  do? 

a.  Tell  the  Soldier  that  the  whole  platoon  will  suffer  if  he/she  doesn't  change. 

b.  Try  to  talk  to  the  Soldier  and  see  if  he/she  has  a  problem  that  you  may  be  able  to  help  with. 

c.  Tell  the  Soldier's  squad  leader. 

d.  Don't  interfere.  It's  up  to  this  Soldier's  squad  or  chain-of-command  to  take  care  of  this. 


Figure  5.  Sample  Criterion  Situational  Judgment  Test  item. 

Job-Related  Attitudes 

We  developed  the  Army  Life  Survey  (ALS)  to  collect  information  from  Soldiers  about 
Army/job-related  attitudes.  The  ALS  measures  satisfaction  with  the  Army  (and  various  facets  of 
the  Army),  organizational  commitment,  perceived  fit  with  the  Army,  perceived  stress,  career 
intentions  (i.e.,  plans  to  attrit  or  reenlist),  and  endorsement  of  the  seven  core  Army  values  (e.g., 
loyalty,  honor,  selfless-service).  Sample  items  are  shown  in  Figure  6.  We  also  developed  the 
Future  Army  Life  Survey  (FALS),  which  asked  Soldiers  to  indicate  how  various  conditions 
expected  to  characterize  the  future  Army  would  affect  their  attitudes  and  career  intentions. 

Attrition 

Some  individuals  who  participated  in  early  research  phases  (e.g.,  in  pilot  or  field  tests)  were 
new  recruits,  and  we  were  able  to  track  them  via  Army  records  and  later  determine  who  left  the 
Army  and  when  they  did  so.  Specifically,  we  examined  (a)  attrition  from  Basic  Combat  Training 
(BCT),  (b)  attrition  from  Advanced  Individual  Training  (AIT),  and  (c)  in-unit  attrition. 
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Instructions:  The  following  survey  includes  several  questions  regarding  your  attitudes,  impressions,  and  thoughts 
about  life  in  the  Army.  Please  read  each  set  of  instructions  and  questions  carefully.  Be  assured  that  your  responses 
will  remain  completely  confidential  and  will  be  used  for  research  purposes  only.  Your  individual  responses  will 
not  be  reported.  It  is  important  that  you  respond  to  each  question  honestly. 

Sample  Items: 

1 .  How  satisfied  are  you  with  the  way  your  supervisor(s)  handle  Soldiers  in  your  unit? 

2.  How  satisfied  are  you  with  the  amount  of  challenge  in  your  work? 

3.  Iam  proud  to  tell  others  that  I  am  in  the  Army. 

4.  I  find  life  as  a  Soldier  to  be  stressful. 

5.  How  confident  are  you  that  you  will  complete  your  current  term  of  service? 


Figure  6.  Sample  Army  Life  Survey  items. 


Considerations  in  Developing  the  Select21  Predictor  Test  Plan 

Valid  pre-enlistment  tests  help  the  Army  select  well-qualified  Soldiers  and  minimize  the 
costs  associated  with  poor  performance.  While  maximizing  validity  of  the  selection  and 
classification  process  is  a  critical  goal,  validity  was  not  the  only  factor  we  considered  in  planning 
Select21  predictor  test  development.  With  this  perspective  in  mind  we  attempted  to  balance 
several  goals  in  deciding  what  types  of  pre-enlistment  measures  to  develop.  These  goals, 
intended  to  maximize  potential  for  operational  implementation,  were  to: 

•  Meet  the  Army’s  future  manpower  needs. 

•  Maximize  the  reliability  and  validity  of  selection  and  classification  decisions. 

•  Minimize  subgroup  differences  and  maximize  fairness  of  the  system. 

•  Minimize  administrative  resource  requirements  for  testing. 

Developing  the  Select21  battery  of  pre-enlistment  measures  required  balancing  all  of  these  goals, 
each  of  which  is  discussed  briefly  below. 

Meeting  the  Army’s  future  manpower  needs  means  ensuring  that  selected  Soldiers  will 
have  the  aptitudes  and  skills  they  need  to  deal  with  the  demands  they  will  face.  Given  that  the 
Army  is  a  promote-from-within  organization,  it  needs  to  select  entry-level  Soldiers  who  have  the 
characteristics  necessary  to  become  successful  leaders.  Additionally,  pre-enlistment  tests  need  to 
identify  Soldiers  who  are  likely  to  be  successful  in  tomorrow’s  Army — given  likely  changes  in 
technology  and  jobs.  Both  of  these  facets  affect  decisions  about  the  characteristics  that  should  be 
measured  by  pre-enlistment  measures.  As  described  earlier,  Select21  conducted  a  systematic, 
future-oriented  job  analysis  to  identify  future  performance  requirements.  The  job  analysis 
resulted  in  pre-enlistment  (KSA)  requirements  that  guided  our  predictor  test  development  plans. 

Maximizing  the  reliability  and  validity  of  pre-enlistment  tests  increases  the  likelihood 
that  the  Army  will  select  Soldiers  who  will  perform  well  in  their  MOS.  Because  several  of  the 
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Select21  predictors  are  assessments  that  do  not  have  “right”  and  “wrong”  answers  like  the 
ASVAB  does,  they  may  be  subject  to  response  distortion.  This  occurs  when  examinees  give  the 
answers  they  believe  the  Army  is  looking  for  rather  than  answers  that  reflect  their  true  beliefs. 
Therefore,  to  maximize  their  validity,  each  of  these  measures  was  designed  to  minimize  the 
potential  for  response  distortion. 

Evidence  of  validity  can  come  from  many  sources — correlations  between  test  scores  and 
job  performance,  linkages  between  job  content  and  characteristics  measured  on  the  tests,  and 
documentation  of  validity  of  the  tests  for  other  similar  jobs  (AERA,  APA,  &  NCME,  1999; 

SIOP,  2003).  Correlational  evidence  is  highly  desirable,  but  also  the  most  expensive  to  obtain. 
Soldiers  must  be  tested  and  rated  by  their  supervisors.  Volumes  of  data  must  be  processed  and 
analyzed.  For  this  reason,  it  was  important  in  Select21  to  invest  resources  in  experimental  pre¬ 
enlistment  measures  that  had  the  best  chance  (based  on  the  job  analysis  information  and  research 
literature)  of  showing  useful  increments  in  predictive  power  over  the  ASVAB  in  a  criterion- 
related  validation  study. 

The  way  in  which  the  criterion-related  validation  investigation  is  conducted  is  important 
to  consider  when  evaluating  the  validity  evidence  it  reveals.  A  concurrent  validation  (i.e., 
involving  experienced  Soldiers)  of  measures  designed  for  administration  to  applicants  in  a  high 
stakes  decision  situation  does  not  necessarily  tell  the  whole  story  particularly  with  regard  to  the 
extent  that  efforts  to  minimize  response  distortion  are  effective. 

Maximizing  the  validity  of  selection  tools  while  minimizing  subgroup  differences  is  a 
challenge  because  cognitive  ability  measures  with  strong  track  records  of  validity  also  typically 
yield  large  differences  among  race  subgroups  (Sackett,  Schmitt,  Ellingson,  &  Kabin,  2001).  One 
way  to  possibly  minimize  the  differences  while  adding  to  the  validity  of  cognitive  measures  is  to 
focus  on  predictors  of  contextual  criteria  (e.g.,  effort  expended  on  the  job).  Based  on  prior 
research,  we  anticipated  that  the  Select21  non-cognitive  predictors  would  yield  smaller  subgroup 
differences  than  those  observed  on  cognitive  aptitude  tests  such  as  the  ASVAB. 

Minimizing  the  administrative  burden  and  costs  of  testing  is  important  for  successful 
implementation  of  pre-enlistment  tests.  Regardless  of  how  it  is  done  (computer-administered, 
proctored  sessions,  group  administration),  testing  costs  time  and  money.  Pre-enlistment  tests  that 
require  a  special  apparatus,  one-on-one  administration,  or  hand  scoring  can  be  overly 
burdensome,  even  if  they  have  useful  validities.  Moreover,  psychometric  goals  are  often  at  odds 
with  administrative  ones.  Longer  tests  are  more  reliable,  but  test  length  increases  testing  time. 

Select21  Experimental  Pre-Enlistment  Tests 

As  mentioned  previously,  the  ASVAB  was  designed  to  predict  technical  performance  in 
training.  There  is  considerable  research  evidence  indicating  the  ASVAB  does  this  quite  well  and 
that  it  also  very  effectively  predicts  technical  aspects  of  job  performance  (Wigdor  &  Green, 
1991).  It  is  less  effective,  however,  for  predicting  the  so-called  “will  do”  aspects  of  performance, 
job  attitudes,  and  attrition  (J.P.  Campbell  &  Knapp,  2001).  Therefore,  the  Select21  research 
focused  on  developing  new  experimental  tests  that  would  complement  the  ASVAB  by  predicting 
these  latter  criteria. 
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The  experimental  Select21  predictors  are  designed  to  measure  temperament,  work-related 
interests  and  values,  judgment,  psychomotor  ability,  and  expectations  about  life  in  the  Army. 
Specifically,  the  experimental  predictor  measures  include  the  following: 

•  Rational  Biodata  Inventory  (RBI) 

•  Work  Suitability  Inventory  (WSI) 

•  Psychomotor  Tests 

•  Predictor  Situational  Judgment  Test  (PSJT) 

•  Record  of  Pre-Enlistment  Training  and  Experiences  (REPETE) 

•  Work  Values  Inventory  (WVI) 

•  Work  Preferences  Survey  (WPS) 

•  Army  Beliefs  Survey  (ABS) 

•  Pre-Service  Expectations  Survey  (PSES) 

•  Army  Work  Knowledge  Survey  (AWKS) 

In  the  remainder  of  this  section,  we  describe  and  provide  examples  for  each  of  the  instruments. 
All  of  the  instruments  are  computer-administered. 

Rational  Biodata  Inventory  (RBI) 

The  Rational  Biodata  Inventory  (RBI)  is  designed  to  measure  motivational/temperament 
characteristics  identified  as  important  to  entry-level  Soldier  performance  in  the  future.  The  RBI 
incorporates  biodata  scales  from  several  existing  Army  tests  designed  to  predict  job  performance 
and  counterproductive  tendencies.  These  existing  tests  are  used,  for  example,  for  selecting 
Special  Forces  Soldiers. 

Biodata  items  ask  about  past  behavior  and  reactions  to  previous  life  events  (e.g.,  “to  what 
extent  have  you  enjoyed  thinking  about  the  plusses  and  minuses  of  alternative  approaches  to 
solving  a  problem?”)-  The  RBI  items  are  written  specifically  to  tap  important  motivational  and 
temperament  characteristics  such  as  the  following: 

•  Achievement  Orientation:  Working  hard  and  giving  one’s  best  effort. 

•  Cognitive  Flexibility:  Willingness  to  try  new  ways  of  getting  work  done.  Embracing 
innovation  and  change.  Comfort  with  uncertainty  and  ambiguity. 

•  Peer  Leadership:  Comfort  with  taking  the  initiative.  Willing  to  make  tough  decisions 
and  accept  responsibility  for  the  group’s  performance. 

•  Cultural  Tolerance:  Willingness  to  accept  and  work  closely  with  those  from  other 
cultural/religious/ethnic  backgrounds. 

Figure  7  shows  an  example  of  a  rational  biodata  item. 
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How  often  have  you  put  off  doing  a  chore  that  you  could  have  taken  care  of  right  away? 

A.  Very  often 

B.  Often 

C.  Sometimes 

D.  Seldom 

E.  Never 


Figure  7.  Example  rational  biodata  item. 


Work  Suitability  Inventory  (WSI) 

The  Work  Suitability  Inventory  (WSI)  uses  a  self-report  format  to  determine  the  types  of 
work  that  respondents  believe  they  would  perform  best.  The  WSI  differs  from  many  other  tests 
in  that  it  asks  respondents  to  sort  statements  rather  than  answer  questions.  Specifically,  this 
measure  asks  respondents  to  sort  16  statements  describing  different  types  of  work  requirements. 
Figure  8  shows  an  example  of  two  of  the  statements  that  appear  on  the  WSI. 


Figure  8.  Screenshot  of partial  sorting  of  statements  from  the  Work  Suitability  Inventory . 
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Each  statement  appears  on  its  own  rectangular  block,  or  “card.”  Respondents  must  rank  the 
16  statements  in  terms  of  how  successful  they  think  they  would  be  at  performing  each  type  of  work 
described — the  highest  ranked  statement  should  describe  work  that  respondents  think  they  would 
be  most  successful  at,  and  the  lowest  ranked  statement  should  describe  work  that  respondents  think 
they  would  be  least  successful  at.  To  rank  the  “type  of  work”  cards,  respondents  use  the  computer 
mouse  to  position  the  cards  on  spaces  labeled  1  (Most  Successful)  through  1 6  (Least  Successful) 
(see  Figure  8).  Compared  with  other  measures,  WS1  takes  very  little  time  to  administer  (less  than 
10  minutes). 

An  important  feature  of  the  WSI  lies  in  its  scoring  approaches.  The  Army  can  score  the 
WSI  differently  for  each  outcome  of  interest  (e.g.,  job  performance,  attrition,  person-Army  fit). 
Unlike  conventional  tests  having  correct  answers  or  a  single  set  of  keyed  answers,  no  single 
ordering  of  the  16  cards  will  likely  predict  all  outcomes  of  interest — a  ranking  that  yields  a  high 
score  on  one  outcome  may  well  yield  a  low  score  on  other  outcomes.  Therefore,  respondents’ 
attempts  to  rank  the  statements  the  way  they  think  the  Army  would  like  (rather  than  ranking 
them  in  the  way  that  best  describes  themselves)  will  not  be  successful.  As  such,  the  WSI  should 
be  resistant  to  response  distortion. 


Psychomotor  Tests 

There  is  good  reason  to  expect  that  tests  of  psychomotor  ability  might  improve  upon  the 
validity  of  the  ASVAB  for  predicting  certain  aspects  of  future  job  performance  in  entry-level 
Army  MOS.  Prior  research  has  shown  psychomotor  tests  to  be  good  predictors  of  gunnery 
performance  and  certain  other  job  performance  criteria  (J.P.  Campbell  &  Knapp,  2001). 
Additionally,  the  Select21  job  analysis  data  suggested  that  psychomotor  skills  were  more  important 
for  performance  in  close  combat  jobs  and  therefore  might  be  useful  for  classification  purposes. 

The  Select21  psychomotor  tests — Target  Tracking  and  Target  Shoot — were  adapted  from 
an  earlier  Army  research  project.  They  are  designed  to  measure  a  person’s  ability  to  adjust  or 
position  a  machine  control  mechanism  precisely.  For  both  tests,  the  respondent  uses  a  joystick  to 
track  a  target. 

Target  Tracking  Test 

On  each  item  of  the  test,  a  path  consisting  of  vertical  and  horizontal  line  segments  is 
presented.  A  sample  item  is  shown  in  Figure  9.  A  target  box  appears  at  the  beginning  of  the  path.  As 
the  item  begins,  the  target  starts  to  move  along  the  path  at  a  constant  rate  of  speed.  The  respondent’s 
task  is  to  keep  the  crosshair  centered  within  the  target  at  all  times  using  a  joystick  to  control 
movement.  The  respondent’s  score  on  this  test  indicates  how  accurately  he  or  she  tracked  the  target 
(i.e.,  the  average  distance  from  the  center  of  the  crosshair  to  the  center  of  target  across  all  items). 
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Figure  9.  Example  Target  Tracking  Test  item. 


Target  Shoot  Test 

At  the  beginning  of  an  item  on  this  test,  a  crosshair  appears  in  the  center  of  the  screen  and 
a  target  box  appears  at  some  other  location  on  the  screen,  as  shown  at  the  top  of  Figure  10.  The 
target  then  begins  to  move  around  the  screen  in  an  unpredictable  manner,  frequently  changing 
direction.  The  respondent  controls  movement  of  the  crosshair  using  a  joystick.  The  respondent’s 
task  is  to  move  the  crosshair  into  the  center  of  the  target  and  press  a  red  button  on  the  joystick  to 
“fire”  at  the  target.  If  successful,  a  “hit”  is  displayed  as  shown  in  the  lower  half  of  Figure  10.  The 
respondent  must  do  this  before  the  time  expires  on  each  item.  The  respondent  receives  two 
scores  on  this  test — a  speed  score  and  an  accuracy  score.  The  speed  score  is  the  average  time 
elapsed  from  the  beginning  of  the  item  until  the  respondent  fired  at  the  target.  The  accuracy 
score  is  the  average  distance  from  the  center  of  the  crosshair  to  the  center  of  the  target  at  the  time 
the  respondent  fired  at  the  target. 
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Figure  10.  Example  Target  Shoot  Test  item. 


Predictor  Situational  Judgment  Test  (PSJT) 

Situational  Judgment  Tests  have  become  popular  in  recent  years  because  they  have  shown 
good  validity  for  predicting  job  performance  and  incremental  validity  over  general  cognitive 
aptitude  measures  like  the  ASVAB.  The  purpose  of  Select21’s  PSJT  is  to  predict  future  first-term 
Soldier  job  performance  by  simulating  situations  youth  face  prior  to  enlistment  as  a  Soldier.  Each 
item  consists  of  a  description  of  a  problem  situation  and  a  list  of  four  alternative  actions  that  a 
person  might  take  in  that  situation.  A  sample  item  is  shown  in  Figure  1 1 . 
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Instructions:  You  are  to  rate  the  effectiveness  of  each  action  using  a  1-7  rating  scale,  where  7  is  highly 
effective  and  1  is  completely  ineffective.  Imagine  that  you  are  in  the  situation;  use  the  scale  to  indicate  how 
effective  or  ineffective  you  believe  each  action  to  be. 

Sample  Item:  You  are  a  student  working  on  a  group  project  with  your  classmates.  You  are  approaching 
your  deadline  and  the  project  is  not  complete.  One  member  of  the  group  offers  to  take  the  project  home 
and  complete  it  over  the  weekend.  How  effective  is  each  of  the  following  actions  you  could  take? 

a.  Offer  to  take  the  project  home  and  finish  it  instead  of  the  other  student  taking  it  home. 

b.  Plan  a  day  for  all  the  group  members  to  get  together  and  work  on  the  project. 

c.  Ask  the  student  if  he/she  needs  help. 

d.  Let  the  student  take  it  home  and  complete  the  project. 

Figure  11.  Example  Predictor  Situational  Judgment  Test  item. 


The  PSJT  measures  a  respondent’s  judgment  in  civilian  situations  that  are  similar  to 
military  situations  experienced  by  Soldiers  during  their  first  few  months  in  the  Army.  During 
development  of  the  test,  NCOs  were  asked  to  write  actual  situations  that  Soldiers  have 
experienced  during  their  first  few  months  in  the  Army.  Next,  Soldiers  in  training  and  civilians 
wrote  civilian  situations  that  paralleled  the  military  situation.  In  most  cases,  the  situation 
involved  either  a  student  at  a  civilian  school  or  an  employee  at  a  civilian  organization.  The  test 
development  process  included  several  rounds  of  writing,  editing,  pilot  testing,  and  analysis. 

Record  of  Pre-Enlistment  Training  and  Experiences  (REPETE) 

Historically,  the  Army  has  provided  training  on  all  required  entry-level  job  skills  for  its 
enlisted  personnel.  By  recognizing  prior  related  training  and/or  experience,  the  Army  could 
reduce  training  requirements  (or  at  least  help  to  ensure  success  in  training).  Additionally, 
applicants  could  benefit  from  their  pre-enlistment  training  and  experience  through  enhanced 
enlistment  options  (in  terms  of  job  choices  and/or  enlistment  bonuses). 

The  development  of  the  Record  of  Pre-Enlistment  Training  and  Experience  (REPETE) 
was  a  demonstration  effort — one  designed  to  illustrate  what  kind  of  instrument  might  be 
developed  and  how  it  might  be  used.  The  demonstration  effort  focused  on  computer  skills,  in 
part  because  although  computer  skills  are  not  included  on  the  ASVAB,  inclusion  of  a  computer 
skills  subtest  has  been  discussed  for  several  years  by  the  Services  and  DoD.  Moreover,  computer 
skills  are  at  least  somewhat  important  for  all  MOS  and  are  particularly  important  for  certain 
MOS  (e.g.,  74B — Information  Systems  Operator/ Analyst). 

The  REPETE  is  a  self-report  measure  designed  to  document  the  type  of  training  and 
experience  that  entry-level  Soldiers  currently  bring  with  them  to  the  Army.  It  focuses  on  10 
computer  skill  categories  that  are  listed  in  Figure  12. 

The  REPETE  has  three  parts,  each  with  multiple  items.  In  the  first  part,  respondents  are 
asked  to  list  courses  they  have  taken  related  to  computer  skills  and  to  indicate  which  of  the  1 0 
computer  skill  categories  were  addressed  in  each  course.  The  second  part  is  structured  similarly 
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but  asks  about  certifications.  The  last  part  asks  respondents  to  rate  themselves  on  each  of  10 
computer  skill  categories  using  a  5-point  level-of-mastery  rating  scale. 


1.  Word  Processing/Desktop  Publishing  Software — Create,  manipulate,  format,  and  print  documents. 

2.  Spreadsheet  Software — Record,  format,  sort,  analyze,  and  graph  information. 

3.  Database  Software — Create,  query,  organize,  analyze,  graph,  and  report  databases. 

4.  Presentation/Graphics  Software — Create  presentation-quality  slides  or  graphics. 

5.  Internet  Usage  and  Information  Search — Send,  receive,  and  save  email;  search  the  Internet. 

6.  Basic  Hardware  and  Operating  Systems — Manage  own  pc  files  and  folders  using  the  operating 
system  and  hardware. 

7.  Networking  and  Computer  Service  Support — Install,  initialize,  configure,  and  manage  network 
software 

8.  Computer  Programming  Principles — Develop  algorithms,  select  programming  languages,  design 
program,  use  assembly  language,  and  develop  documentation. 

9.  Basic  Web  Programming — Develop  web  pages  using  HTML  and  Javascript. 

10.  Object-Oriented  Programming  Concepts — Create  Object-Oriented  Modeling  using  UML  notation; 
use  C++,  Java,  or  Visual  Basic. 

Figure  12.  Computer  skills  categories  used  in  the  REPETE. 


Because  the  REPETE  is  not  a  predictor  in  the  traditional  sense  and  because  it  would 
require  Soldiers  to  try  to  recall  their  pre-enlistment  status  with  regard  to  the  questions,  it  was  not 
included  in  the  concurrent  validation  effort.  With  additional  research,  however,  the  REPETE 
could  be  expanded  and  used  to  give  credit  for  a  broad  range  of  pre-enlistment  skills  and 
experience.  Indeed,  we  currently  are  adding  a  section  on  language  skills  for  the  Army  Class 
project.  Such  a  tool  would  supplement  the  current  Army  Civilian  Acquired  Skills  Program 
(ACASP),  which  already  offers  enlistment  bonuses,  accelerated  promotions,  and/or  abbreviated 
training  for  those  enlistees  who  meet  certain  MOS-specific  skills  or  certification  criteria. 

Work  Values  Inventory 

People  tend  to  value  different  things  when  it  comes  to  work  (e.g.,  physical  activity, 
autonomy).  Similarly,  organizations  tend  to  vary  in  the  degree  to  which  they  offer  characteristics  of 
work  that  individuals  value.  The  purpose  of  the  Work  Values  Inventory  is  to  assess  respondents’ 
preferences  for  various  characteristics  of  work  (e.g.,  On  my  ideal  job,  I  would  make  decisions  on  my 
own.).  It  is  based  on  the  premise  that  not  all  respondents  value  characteristics  of  work  offered  by  the 
Army  and  that  a  disconnect  between  what  respondents  value  and  what  the  Army  offers  can  lead  to 
undesirable  outcomes.  Specifically,  past  research  suggests  that  individuals  who  value  work 
characteristics  that  their  employer  provides  are  more  satisfied  and  more  likely  to  remain  in  the 
organization  than  those  who  value  characteristics  not  provided  by  their  organization. 

The  WVI  yields  an  ordering  of  28  work  characteristics  in  terms  of  how  important  they 
would  be  in  their  ideal  job,  and  distinguishes  between  important  and  unimportant  characteristics 
(in  an  absolute  sense).  To  make  the  rank  ordering  activity  easier,  respondents  are  first  asked  to 
sort  the  28  work  characteristics  into  four  categories  of  varying  importance  (A-D).  For  example, 
respondents  place  their  seven  most  important  work  characteristics  in  Category  A  and  their  seven 
least  important  work  characteristics  in  Category  D.  Respondents  then  rank  order  the  importance 
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of  the  work  characteristics  within  each  category.  After  completing  their  rankings  within  each 
category,  respondents  are  presented  with  the  full  list  of  28  work  characteristics  in  the  order  they 
ranked  them.  Respondents  are  asked  to  review  the  list  and  to  make  a  line  through  it  such  that  the 
work  characteristics  important  for  their  ideal  job  are  above  the  line  and  the  characteristics  below 
the  line  are  unimportant  to  have  on  their  ideal  job.  Once  a  respondent  completes  the  WVI,  the 
Army  can  assess  the  similarity  of  his  or  her  preferences  for  various  work  characteristics  with 
what  the  Army  actually  offers. 


Work  Preferences  Survey  (WPS) 

Occupations  can  be  categorized  into  six  types:  Realistic,  Investigative,  Artistic,  Social, 
Enterprising,  and  Conventional  (Holland,  1997).  Individuals  vary  in  the  degree  to  which  they  are 
interested  in  these  types  of  occupations.  Research  has  shown  that  people  are  drawn  to 
occupations  that  they  think  will  satisfy  their  interests  (e.g.,  D.P.  Campbell  &  Hansen,  1981; 
Strong,  1955).  For  example,  people  who  are  creative  and  expressive  tend  to  seek  out  Artistic 
occupations.  Studies  also  have  found  that  a  good  match  between  an  individual’s  interests  and  the 
interests  that  his  or  her  chosen  occupation  supports  can  lead  to  a  variety  of  positive  outcomes  for 
both  the  worker  and  the  organization.  For  the  worker,  a  good  match  in  terms  of  interests  can 
result  in  higher  job  satisfaction  (e.g.,  Tranberg,  Slane,  &  Ekeberg,  1993).  For  the  organization, 
more  satisfied  workers  tend  to  work  harder  and  are  less  likely  to  leave  (e.g.,  Griffeth,  Horn,  & 
Gaertner,  2000;  Judge,  Thorenson,  Bono,  &  Patton,  2001). 

The  Work  Preferences  Survey  is  designed  to  assess  applicants’  preferences  for  each  of 
Holland’s  six  interest  areas.  Respondents  are  asked  to  rate  their  interest  in  various  work 
activities,  work  environments,  and  learning  opportunities  that  reflect  these  occupations.  The 
Army  can  then  compare  the  interests  of  each  respondent  to  the  interests  the  Army  supports  to 
assess  the  match  between  the  two.  Figure  13  shows  the  instructions  for  the  instrument  and  three 
items  that  are  similar  to  those  that  appear  on  the  actual  survey. 


Instructions:  The  purpose  of  this  survey  is  to  find  out  what  your  ideal  job  would  be  like.  Please  read  each 
statement.  Then,  using  the  scale  below,  indicate  how  important  each  statement  is  to  your  ideal  job. 


1 

2 

3 

4 

5 

Extremely 
unimportant  to 
have  in  my  ideal 
job 

Not  important  to 
have  in  my  ideal 
job 

Not  important  or 

unimportant  to 
have  in  my  ideal 
job 

Important  to  have 
in  my  ideal  job 

Extremely 
important  to  have 
in  my  ideal  job 

Sample  Statements 

1 .  A  job  that  requires  me  to  repair  broken  machinery. 

2.  A  job  in  which  I  work  by  myself  most  of  day. 

3.  A  job  in  which  I  can  learn  more  about  running  a  business. 


Figure  13.  Sample  Work  Preferences  Survey  items. 
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Expectations  Measures 


The  last  three  experimental  measures  are  based  on  the  premise  that  not  all  applicants  have 
accurate  beliefs  about  what  it  is  like  to  be  in  the  Army  and  that  inaccurate  beliefs  can  lead  to 
undesirable  outcomes.  Specifically,  past  research  suggests  that  people  who  have  accurate  beliefs 
about  the  organization  to  which  they  are  applying  tend  to  be  more  satisfied  and  less  likely  to 
attrit  than  those  with  less  accurate  beliefs.  Each  of  the  three  instruments — Army  Beliefs  Survey, 
Pre-Service  Expectations  Survey,  and  Army  Work  Knowledge  Survey — is  based  on  content  from 
previously  described  measures.  In  each  of  these  measures,  respondents  are  asked  to  indicate  the 
extent  to  which  each  statement  is  descriptive  of  the  Army  environment.  Responses  are  compared 
to  the  answers  given  by  a  sample  of  experienced  NCOs.  Because  the  items  concern  pre-service 
knowledge,  they  are  not  appropriate  for  administration  to  people  already  in  the  Army;  therefore, 
they  were  not  included  in  the  Select21  concurrent  validation  data  collection. 

Army  Beliefs  Survey  (ABS) 

The  Army  Beliefs  Survey  (ABS)  assesses  respondents’  knowledge  about  what  work  in 
the  Army  involves.  The  content  of  the  ABS  parallels  the  content  of  the  Work  Values  Inventory. 
Figure  14  shows  examples  of  what  items  on  this  survey  are  like.  When  respondents  complete  this 
survey,  the  Army  can  assess  the  similarity  of  their  beliefs  of  Army  life  with  the  reality  of  Army 
life  for  first-term  Soldiers. 


Instructions:  Using  the  rating  scale  below,  indicate  which  category  you  believe  best  describes  each 
statement. 


Few  Will  Experience 

Some  Will  Experience 

Most  Will  Experience 

This  statement  describes  an 
experience /ew  Soldiers  will 
have  during  their  first 
enlistment. 

This  statement  describes  an 
experience  some,  but  not  most, 
Soldiers  will  have  during  their 
first  enlistment. 

This  statement  describes  an 
experience  most  Soldiers  will 
have  during  their  first 
enlistment. 

Sample  Statements 

1 .  Soldiers  will  improve  their  physical  fitness. 

2.  Soldiers  will  make  decisions  on  their  own. 


Figure  14.  Sample  Army  Beliefs  Survey  items. 


Pre-Service  Expectations  Survey  (PSES) 

The  interest  measure  described  previously  (the  Work  Preferences  Survey)  assesses  the 
respondents’  degree  of  interest  in  each  of  six  occupational  areas  (e.g.,  Artistic,  Social).  In  contrast, 
the  Pre-Service  Expectations  Survey  (PSES)  assesses  the  extent  to  which  respondents  believe  the 
Army  will  provide  work  activities  and  training  opportunities  associated  with  each  area.  The  Army 
can  then  determine  how  closely  respondents’  expectations  match  the  realities  of  Army  work. 
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Individuals  completing  the  PSES  are  asked  to  read  a  brief  description  of  each  of  the  six 
work  settings  listed  above  and  then  rate  the  extent  to  which  they  think  each  type  describes  the 
Army  work  environment. 

Army  Work  Knowledge  Survey  (A  WKS) 

The  Army  Work  Knowledge  Survey  (A WKS)  lists  the  same  16  statements  that  appear  on 
the  Work  Suitability  Inventory.  In  contrast  to  the  WSI,  respondents  on  the  AWKS  indicate  the 
extent  to  which  they  agree  with  each  statement  using  a  5-point  scale.  Following  is  a  sample 
AWKS  statement:  “Work  as  a  first  term  Soldier  in  the  Army  often  requires  showing  a 
cooperative  and  friendly  attitude  towards  others  I  dislike  or  disagree  with.” 

What  Do  We  Know  about  These  Tests  and  What  Else  are  We  Asking? 

We  have  summarized  the  criteria  for  evaluating  pre-enlistment  selection  and 
classification  tests  and  described  the  experimental  tests  that  were  developed  in  the  Select2 1 
project.  In  this  section,  we  describe  what  have  we  learned  about  these  tests  and  about  the 
ASVAB. 


What  We  Know 

Our  intent  was  to  develop  predictors  that  supplement  the  ASVAB  for  the  prediction  of 
performance  and  attitudinal  criteria.  We  constructed  five  composite  performance  scores  and  five 
attitudinal  scores  to  use  in  the  validation  analyses.  Information  from  the  rating  scales,  job 
knowledge  tests,  criterion  situational  judgment  test,  and  the  Army  Life  Survey  was  used  to 
generate  five  performance  criterion  scores:  (a)  General  Technical  Proficiency,  (b)  Achievement 
and  Effort,  (c)  Physical  Fitness,  (d)  Teamwork,  and  (e)  Future  Expected  Performance.  We  also 
used  five  attitudinal  scores  generated  from  the  Army  Life  Survey  and  the  Future  Army  Life 
Survey:  (a)  Satisfaction  with  the  Army,  (b)  Perceived  Army  Fit,  (c)  Attrition  Cognitions,  (d) 
Career  Intentions,  and  (e)  Future  Army  Affect. 

Entry-Level  Selection  Decisions 

Consistent  with  years  of  prior  research,  scores  on  the  ASVAB  were  shown  to  be  good 
predictors  of  can-do  performance  criteria  (e.g..  General  Technical  Proficiency),  but  have  less 
validity  for  predicting  will-do  (e.g.,  Physical  Fitness,  Teamwork)  and  attitudinal  criteria.  The 
ASVAB  predicted  General  Technical  Proficiency  so  well  that  it  left  relatively  little  room  for 
improvement  (“incremental  validity”)  with  the  addition  of  other  tests.  Despite  this  finding, 
several  of  the  experimental  predictors  showed  small  but  statistically  significant  (and 
operationally  useful)  improvements  over  the  ASVAB  alone  for  predicting  General  Technical 
Proficiency  scores. 

On  the  other  hand,  many  of  the  Select21  predictors  showed  notable  levels  of  incremental 
validity  over  the  ASVAB  when  predicting  Achievement  and  Effort,  Physical  Fitness,  and 
Teamwork  performance.  Such  findings  reinforce  the  notion  that  when  judging  the  efficacy  of 
predictors  for  incrementing  the  validity  of  the  ASVAB,  it  is  important  to  account  for  the  multi- 
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dimensional  nature  of  the  performance  and  attitudinal  criteria  the  Army  is  interested  in 
predicting. 

Substantial  levels  of  incremental  validity  were  found  for  the  RBI,  WVI,  and  WPS  for 
predicting  the  attitudinal  criteria,  with  somewhat  lower  levels  of  validity  for  the  WS1  and  PSJT. 
ASVAB  scores  also  yielded  significant  correlations  with  future  expected  performance  scores. 

This  is  a  new  finding  and  one  that  bears  emphasis — ASVAB  has  been  predictive  of  “can  do” 
performance  historically  and  now  we  have  evidence  that  it  should  continue  to  be  useful  well  into 
the  Army’s  future.  We  also  learned  that  the  pattern  of  validity  findings  when  looking  at  current 
versus  future  criteria  was  similar,  suggesting  that  it  is  not  necessary  (at  least  at  this  point)  to 
continue  to  try  to  measure  future  criteria  in  this  program  of  validation  research. 

In  addition  to  the  criterion-related  validity  evidence,  the  Select21  predictors  demonstrate 
sufficient  content  validity.  That  is,  the  measures  were  developed  to  assess  psychological 
constructs  that  either  emerged  from  the  job  analysis  work  or  which  were  suggested  by  a  large 
body  of  research  on  person-environment  fit.  More  reliance  on  content  validity  evidence  was  one 
of  the  recommendations  of  the  ASVAB  technical  review  panel  (Drasgow  et  al.,  2006). 

Entry-Level  Personnel  Classification  Decisions 

Currently,  the  nine  operational  ASVAB  tests  are  combined  in  various  ways  to  compute 
Aptitude  Area  scores  that  are  used  to  classify  new  recruits  into  MOS.  For  the  most  part,  these 
composites  are  based  on  criterion-related  validity  evidence  that  is  becoming  very  dated  because  the 
primary  source  of  data  was  Skill  Qualification  Test  (SQT)  scores  that  ceased  being  collected  in  the 
early  1990s.  The  Army  is  sponsoring  research  not  only  to  identify  new  predictor  tests  that  could 
contribute  to  classification  decisions  but  also  to  update  the  empirical  research  on  which  the 
ASVAB  Aptitude  Area  composites  are  based. 

As  mentioned  earlier  in  this  report,  we  did  not  collect  enough  MOS-specific  criterion  data 
to  permit  direct  evaluation  of  the  potential  for  the  Select21  predictor  tests  to  support  effective 
classification  decisions.  We  also  did  not  have  enough  participants  from  individual  MOS  to 
examine  prediction  of  Army-wide  criteria  at  that  level.  Instead,  we  performed  subgroup  analyses 
using  MOS  type  as  the  subgrouping  variable  to  get  an  idea  of  the  potential  for  the  experimental 
predictors  to  improve  classification  efficiency.  Soldiers  were  sorted  into  four  clusters  of  MOS  for 
these  analyses,  which  did  suggest  that  some  of  the  predictors  have  potential  utility  for 
classification.  Four  predictors  had  scores  that  showed  differences  in  validity  estimates  across 
clusters  for  three  or  more  criterion  composites:  the  RBI,  WSI,  WPS,  and  Target  Tracking.  Other 
predictors  showed  more  targeted  results  focused  on  specific  cluster  comparisons  or  criteria. 

What  Else  are  We  Asking? 

How  will  the  Tests  Work  with  Applicants? 

Select21’s  concurrent  validation  research  design  fundamentally  differs  from  an  operational 
setting  in  which  predictors  would  be  administered  to  applicants  instead  of  experienced  Soldiers. 
There  are  several  ways  in  which  one  might  expect  findings  from  a  concurrent  design  to  differ  from 
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an  operational  setting,  but  here  we  focus  on  two  factors  that  are  of  particular  concern  in  the 
Select21  research — (a)  the  response  distortion  that  is  likely  to  occur  when  non-cognitive  measures 
such  as  the  RBI,  WPS,  and  WVI  are  administered  to  applicants  and  (b)  contaminant  variation  in 
predictor  measures  arising  from  their  administration  to  experienced  Soldiers. 

With  regard  to  the  response  distortion  issue,  Soldiers  participating  in  a  research  effort 
have  little  motivation  to  make  themselves  look  appealing  to  the  Army  in  their  responses  to 
experimental  measures  because  they  know  that  the  scores  will  not  be  used  to  make  decisions 
about  them.  However,  respondents  who  have  not  yet  enlisted  will  be  more  motivated  to  look 
good  and  one  can  expect  at  least  some  applicants  to  be  coached  on  how  to  do  well  on  the  pre¬ 
enlistment  screening  tests.  The  extent  to  which  the  effectiveness  of  the  Select21  self-report 
temperament  and  interest  measures  would  be  compromised  in  an  operational  environment  needs 
to  be  further  investigated. 

Another  factor  that  may  affect  the  generalizablity  of  the  concurrent  validation  results  is  that 
Soldiers’  responses  to  predictor  measures  may  be  influenced  by  the  experiences  they  have  gained 
in  the  Army.  For  example,  many  of  the  items  on  the  RBI  ask  about  past  behavior,  but  for 
experienced  Soldiers,  this  includes  post-enlistment  behaviors  likely  influenced  by  the  fact  they 
have  been  in  the  Army.  In  applicant  samples,  respondents  can  only  answer  RBI  items  based  on 
“pre-Army”  behavior.  Another  example  of  this  phenomenon  occurs  with  the  WSI  which  asks 
Soldiers  what  types  of  work  they  think  they  would  be  able  to  perform  best.  Their  answers  may  be 
influenced  by  their  Army  experience.  Moreover,  there  were  several  Select2 1  predictors  that  we  did 
not  include  in  the  concurrent  validation  at  all  because  it  did  not  make  sense  to  administer  them  to 
experienced  Soldiers  (e.g.,  the  Army  Beliefs  Survey,  the  Pre-Service  Expectations  Survey).  A 
longitudinal  design  is  needed  in  which  measures  are  administered  to  applicants  or  new  recruits. 

How  will  the  Tests  Work  for  Classification  Decision-Making?  How  Should  the  Tests  be  Scored 
for  Classification  Purposes? 

Another  open  question  is  the  classification  efficiency  of  the  experimental  predictors. 
Although  Select2 1  yielded  evidence  suggesting  several  predictors  have  potential  to  improve 
classification  decisions,  more  direct  evidence  is  needed.  With  regard  to  classification 
applications,  the  accepted  method  for  determining  the  precise  way  for  scoring  the  ASVAB  or 
other  tests  for  classification  purposes  involves  examining  the  relationships  of  test  scores  to 
MOS-specific  criterion  data.  Given  the  large  number  of  MOS,  collecting  the  requisite  criterion 
data  needed  to  support  this  traditional  approach  is  infeasible.  Thus,  yet  another  question  is  how 
can  researchers  provide  the  information  needed  to  update  the  ASVAB  Aptitude  Area  scores  and 
operationalize  new  classification  tests? 

Army  Class  Project  to  Provide  Answers 

The  Army  has  begun  to  address  these  questions  in  the  follow-on  Army  Class  project.  Of 
particular  relevance  are  three  elements  of  this  new  project  which  are  presented  below  in 
chronological  order. 

•  Collection  of  more  concurrent  validation  data. 
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•  Development  of  alternate  classification  research  strategies. 

•  Collection  of  longitudinal  validation  data. 

Army  Class  researchers  have  collected  additional  concurrent  validation  data  from 
Soldiers  in  five  target  MOS.  Combined  with  the  MOS-specific  data  collected  from  1  IB  and  25U 
Soldiers  in  Select21 ,  this  will  allow  for  a  more  direct  evaluation  of  the  experimental  predictors’ 
utility  for  supporting  classification  decisions. 

A  six-member  panel  of  leading  technical  experts  was  established  in  the  Army  Class 
project  to  identify  realistic  but  effective  strategies  for  conducting  future  classification  research.  It 
is  doubtful  that  they  will  be  able  to  offer  strategies  that  eliminate  the  need  for  MOS-specific 
criterion  data,  but  it  is  expected  they  will  offer  strategies  for  minimizing  this  requirement. 

Finally,  the  follow-on  project  will  include  a  longitudinal  validation  that  will  involve 
administering  the  most  promising  experimental  predictors  to  either  Army  applicants  or  new 
recruits  and  then  tracking  those  individuals  through  their  first  term  of  service  to  collect 
performance  and  attrition  data.  We  expect  that  these  findings  will  provide  a  more  accurate 
forecast  of  how  effective  the  predictors  will  be  in  an  operational  environment. 

What  Else  Should  We  be  Asking? 

There  are  several  questions  of  interest  that  would  benefit  from  further  research,  but  for 
which  such  research  is  not  currently  planned.  They  are  offered  here  as  suggestions  for  the 
Army’s  future  research  agenda. 

Criterion  Policy 

“An  organization’s  choice  of  criteria  for  personnel  research  significantly  affects 
how  research  results  will  influence  the  design  of  the  selection  and  classification 
system.  In  effect,  criterion  policy  reflects  the  organization’s  intended  definition 
for  effective  performance  in  that  organization,  and  the  types  of  predictors  that  are 
used  in  selection  and  classification  decision  making  will  depend  upon  the  criteria 
against  which  they  are  compared.  Systematic  consideration  of  criterion  policy  is 
necessary  so  that  informed  decisions  can  be  made  about  future  predictor  and 
criterion  development”  (J.P.  Campbell,  Russell,  &  Knapp,  1994). 

Findings  in  Project  A  (J.P.  Campbell  &  Knapp,  2001),  NC021  (Knapp,  McCloy,  & 
Heffner,  2004),  and  Select21  (Knapp  &  Tremble,  2006)  all  confirm  that  criteria  matter  and  that 
validation  results  differ  substantially  by  the  criteria  of  choice.  Recent  research,  supporting  policy 
decisions,  has  sought  to  go  beyond  indices  of  training  success  by  using  SQT  scores  to  validate 
ASVAB  classification  composites.  SQT  scores  can  be  thought  of  as  measures  of  technical 
proficiency,  and  the  choice  of  their  use  in  validation  efforts  could  be  taken  to  mean  that  MOS 
technical  proficiency  is  the  most  important  (if  not,  only)  criterion  for  prediction  by  selection  and 
classification  personnel  tests.  However,  future  criterion  policy  issues  facing  the  Army  have  to  do 
with  how  overall  job  performance  is  construed,  measured,  and  used  in  the  selection  and 
classification  context.  Is  there  a  consensus  within  the  Army  about  the  goals  of  criterion 
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measurement?  Should  job  performance  be  construed  more  broadly  (perhaps  reflecting 
organizational  fit)  and  include  non-technical  aspects  of  job  performance  such  as  the  individual’s 
effort  and  achievement  or  ability  to  work  with  a  team?  Should  prediction  of  such  non-technical 
aspects  of  job  performance  play  a  more  important  role  in  selection  and  classification  decisions? 
Research  on  criterion  policy,  conducted  in  collaboration  with  policymakers,  could  be  used  to 
develop  or  identify  consensus.  Moreover,  decisions  by  policymakers  about  the  merits  of  various 
criteria  could  be  used  to  guide  funding  and  resource  allocations. 

Selection  and  Classification  Algorithms 

Once  a  decision  is  made  to  include  multiple  criteria  in  selection  and  classification 
research,  there  are  many  ancillary  questions  to  be  addressed.  How  will  it  be  implemented?  How 
can  the  maximum  potential  gain  from  classification  be  estimated  given  that  there  are  choices 
among  predictor  batteries,  performance  goals,  and  criterion  measurement  methods?  There  are  a 
large  number  of  permutations  of  predictors,  criteria,  and  goals.  How  can  we  efficiently  simulate 
the  outcomes  of  different  predictor/criterion/goal  combinations?  How  successfully  can 
operational  job  assignment  procedures  capture  the  potential  classification  gains? 

Forecasting  the  Future 

An  important  contribution  of  the  Select21  research  and  the  preceding  NC021  research 
was  the  emphasis  on  preparing  Soldier  selection  and  promotion  systems  to  meet  future  needs. 
This  focus  is  particularly  relevant  to  military  organizations  that  promote  from  within,  but  this 
focus  is  rare  in  either  military  or  civilian  settings.  Moreover,  our  observation  that  Army  future 
planning  documents  rarely  contain  information  about  implications  of  new  weapons  systems  and 
other  changes  for  individual  Soldiers  is  unfortunate,  given  that  Soldiers  are  the  ones  who  must  be 
able  to  operate  effectively  in  the  environment  that  planners  are  trying  to  create.  In  any  case,  a 
risk  in  forecasting  the  future  is  that  the  projections  may  not  be  accurate.  Indeed,  without  a  crystal 
ball,  it  must  be  assumed  that  some  details  are  likely  going  to  be  off  the  mark. 

Thus,  we  see  at  least  two  areas  for  additional  research.  One  would  be  evaluating  the 
accuracy  of  the  forecasts  that  have  been  made  in  the  Select21  and  NC021  research  programs, 
particularly  as  they  regard  the  conditions  under  which  enlisted  personnel  will  be  operating  in  the 
next  1 5-20  years.  Understanding  where  those  predictions  were  accurate  and  where  they  were  not 
could  help  hone  the  method  for  conducting  future-oriented  job  analysis  work.  Second,  it  would 
also  make  sense  to  establish  a  research-based  procedure  for  periodically  assessing  future 
conditions  so  the  Army  can  continue  to  stay  ahead  of  the  curve  in  meeting  future  personnel 
requirements. 
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From  Research  to  Implementation 

If  research  is  to  make  a  difference  in  an  operational  or  user  system,  the  research,  no 
matter  how  innovative,  must  address  and  inform  relevant  policy  and  operational  issues.  This  is  as 
true  for  military  personnel  research  and  as  it  is  for  research  conducted  in  the  civilian  sector. 

Thus,  if  research  is  to  make  a  difference,  it  needs  help  in  transferring  from  the  laboratory  to  the 
operational  environment.  What  is  needed  is  an  “implementation”  perspective — an  approach  that 
focuses  on  actions  that  convert  research  into  policy  and  operational  procedures.  In  the  remainder 
of  this  report,  we  broaden  our  discussion  to  include  all  military  personnel  research  programs,  not 
just  the  Select21  effort. 


What  is  Implementation? 

Implementation  can  be  defined  as  a  specified  set  of  activities  intended  to  translate 
research  findings  into  informed  policies  and  operational  practices.  In  other  words, 
implementation  means  putting  into  place  new  policies  and  procedures  with  the  adoption  of 
research  as  the  basis  for  those  policies  and  procedures.  In  many  cases,  effective  implementation 
is  a  bigger  challenge  than  conducting  innovative  research.  Desirable  policy  and  operational 
outcomes  are  likely  achieved  only  when  the  research  includes  activities  intended  to  translate 
research  findings  into  informed  policies  and  operational  practices.  Research  plans  and  activities 
need  to  ensure  that  the  research  is  well  connected  to  the  operational  world  both  in  terms  of  the 
users’  needs  for  research  innovations  and  in  terms  of  where  and  how  the  innovative  outcomes  fit 
and  can  be  placed  into  the  user’s  operational  systems. 

The  Role  of  Researchers  and  Policymakers 

Projects  of  high  impact  and  value  such  as  Select21  must  be  targeted  from  the  beginning 
for  aggressive  implementation.  To  that  end,  researchers,  policymakers,  and  other  stakeholders 
together  must  plan  and  then  broker  implementation  of  research  into  the  Army’s  selection  and 
classification  system.  Researchers  must  be  connected  to  the  operational  world,  asking  research 
questions  that  reflect  their  appreciation  for  and  understanding  of  its  complexity.  Researchers  and 
policymakers  must  have  two-way  communications.  Each  group  must  value  and  listen  to  input 
from  the  other. 

For  example,  for  researchers  to  design  and  conduct  meaningful  and  effective  selection  and 
classification  research,  they  must  be  knowledgeable  about  recruiting  and  enlistment  processing 
policies  and  procedures.  From  the  earliest  stages  of  research  planning,  researchers  should  establish 
rapport  with  relevant  policymakers  and  stakeholders  with  a  focus  on  learning  more  about 
contemporary  issues  such  as  recruit  quality  requirements,  filling  difficult  and  demanding  MOS,  and 
attrition-reducing  initiatives.  In  the  case  of  Select21,  the  research  has  had  the  added  requirement  of 
understanding  the  shape  of  these  issues  in  the  future.  This  rapport  is  critical  for  research  on 
innovations  that  users  want.  This  rapport  and  the  prospect  of  useful  products  encourage  policymakers 
to  become  liaisons  for  fitting  research  outcomes  into  the  operational  system. 
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Army  and  Joint-Service  Players 


In  the  context  of  Select21  research,  there  are  several  groups  of  policymakers  and  other 
stakeholders  who  should  be  involved  in  designing  the  research  and  planning  for  its 
implementation.  Within  the  Army,  policymakers  include  officials  from  the  Offices  of  the 
Assistant  Secretary  of  the  Army  for  Manpower  and  Reserve  Affairs,  the  Army  G-l ,  and  the 
Army  Training  and  Doctrine  Command  (TRADOC).  Other  stakeholders  comprise  personnel 
from  subordinate  organizations  such  as  the  various  TRADOC  training  schools,  the  Army 
Accessions  Command,  the  Army  Recruiting  Command,  and  the  Human  Resources  Command. 
Adding  to  this  complexity  is  the  fact  that  while  each  Service  has  its  own  recruiting  procedures 
and  enlistment  standards.  Army  personnel  selection  and  classification  does  not  operate  in  a 
vacuum.  Many  aspects  of  the  military  enlistment  process  are  common  across  the  Services.  To  the 
extent  the  Army’s  system  touches  the  other  Services,  changes  to  its  selection  and  classification 
process  must  be  coordinated  with  a  variety  of  players  within  a  Joint-Service  environment.  These 
players  include  officials  in  the  Office  of  the  Secretary  of  Defense  (OSD),  the  other  Services,  and 
elements  within  DoD  such  as  the  US  Military  Entrance  Processing  Command  (USMEPCOM). 

To  facilitate  cooperation  and  coordination,  the  Joint-Service  Manpower  Accession  Policy 
Working  Group  (MAPWG)  addresses  inter-service  testing  and  enlistment  processing  issues  and 
passes  its  deliberations  to  upper  echelons  for  decision.  The  MAPWG  comprises  policy  staff 
members  from  OSD  and  each  of  the  Services  as  well  as  technical  representatives  from  the 
Defense  Manpower  Data  Center,  USMEPCOM,  and  the  Service  personnel  research 
establishments. 

Developing  and  maintaining  relationships  for  implementation  is  a  special  challenge  in 
military  organizations.  Characteristic  of  such  organizations  is  the  periodic  transfer  of  military 
personnel,  to  include  senior  military  officers  in  policymaking  positions.  Policymakers  have 
individual  and  career  experiences  that  can  lead  to  differences  in  the  ways  they  consider  the  need 
for  research.  As  experience  in  various  assignments  is  gained,  knowledge  and  skill  improvements 
are  accumulated  by  policymakers.  When  one  policymaker  is  replaced  by  another,  the 
accumulated  knowledge  is  lost,  and  the  perspective  on  research  and  implementation  is  altered  to 
reflect  the  philosophy,  approach,  experience,  and  skills  of  the  new  policymaker.  Moreover,  the 
new  policymaker  is  likely  to  be  confronted  with  rapidly  changing  issues  endemic  to  the  military 
environment.  Consequently,  policymakers  may  not  fully  appreciate  the  importance  of  ongoing 
research  since  it  may  pertain  mainly  to  issues  of  importance  to  their  predecessors.  Thus,  it  is 
incumbent  on  the  researcher  to  establish  a  relationship  with  the  new  policymaker,  to  refine  the 
relevant  issues,  to  demonstrate  how  the  research  will  positively  affect  the  Army,  and  to  develop 
mutual  confidence  and  trust. 


Select21  Implementation  Activities 

To  define  products  with  good  chances  of  meeting  user  needs,  Select21  was  designed  to 
take  into  account  the  history  of  earlier  research  projects.  This  history  had  shown  the  strengths  of 
the  ASVAB,  supported  inferences  about  the  types  of  measures  that  would  likely  be  most  useful 
within  this  system,  and  highlighted  methodological  issues  that  needed  to  be  addressed  prior  to 
transition  into  the  operational  setting.  More  specifically,  this  history  was  a  basis  for  research 
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designed  to  (a)  produce  non-cognitive  measures  for  augmenting  the  current  ASVAB  that  would 
resist  distortion  in  high-stakes  testing  situations  and  (b)  examine  the  measures  against  a 
performance  or  criterion  space  that,  in  addition  to  technical  job  performance,  included 
motivational  and  other  outcomes  indicative  of  an  individual’s  fit  into  the  organization.  In  other 
words,  the  goals  for  Select21  were  to  produce  candidate  measures  and,  in  addition,  evidence  on 
their  promise  if  integrated  into  the  current  enlisted  personnel  selection  and  classification  system. 

Dialogue  with  Army  stakeholders  was  an  integral  feature  of  the  research  method.  The 
earlier  NC021  research  had  shown  that  a  future-oriented  job  analysis  requires  intense 
involvement  of  a  few  people  with  specific  expertise  and  that  individuals  highly  knowledgeable 
about  the  Army’s  future  plans  are  not  necessarily  knowledgeable  about  the  specifics  of  jobs  and 
vice  versa.  In  Select21,  SMEs  were  carefully  selected  to  provide  the  varying  types  of  expertise 
needed,  and  they  participated  in  a  series  of  workshops  to  develop  and  review  materials.  These 
workshops  provided  Select21  the  needed  SME  input  but  also  made  the  project  visible  to 
stakeholders  in  offices  concerned  with  the  emerging  future  Arm\  and  with  proponency  for 
training  Soldiers  in  the  skills  needed  in  operational  units. 

Many  of  the  SME  reviews  took  place  in  the  context  of  meetings  of  two  Army  panels  that 
remained  involved  with  the  research  over  the  course  of  the  future-oriented  job  analysis  and 
through  the  initial  phases  of  data  collection  on  the  Select21  measures.  One  panel  was  an  Army 
Steering  Committee.  The  Steering  Committee  was  a  policy  advisory  group  composed  of  senior 
representatives  from  a  number  of  stakeholder  organizations  concerned  with  the  transformation 
toward  the  Army’s  future  force.  One  of  its  roles  was  to  provide  top-down  information  and 
direction  to  ensure  that  Select21  plans  and  products  remained  consistent  with  the  vision  and 
needs  of  the  Army.  The  second  group,  a  senior  Subject  Matter  Expert  Panel  (SMEP),  was 
composed  of  personnel  expert  in  particular  MOS  or  in  plans  for  specific  aspects  of  the  future. 
SMEP  members  provided  essential  bottom-up  information  about  particular  MOS  and  top-down 
information  about  ongoing  efforts  focusing  on  the  Army’s  transformation.  The  SMEP  worked 
closely  with  the  project  team  on  specific  research  products  and  troop  support  for  workshops  and 
other  data  collections. 

Communications  also  occurred  with  Army  stakeholders  who  were  not  connected  directly 
to  Select21  through  the  Steering  Committee  or  the  SMEP.  This  communication  largely  sought  to 
stimulate  support  for  continuation  of  research  that  requires  a  relatively  long-term  time  horizon. 
The  communication  included  briefings  of  top-level  policymakers  concerned  with  enlistment 
procedures,  various  research  review  panels,  and  system  operators  taking  part  in  conferences  and 
working  groups  regarding  relevant  issues.  The  Select21  research  team  also  devised  a  type  of 
report  intended  to  describe  research  outcomes  and  encourage  interest.  These  reports,  called 
product  reports,  focused  on  the  achievement  to  give  potential  sponsors  and  users  a  sense  of 
progress  toward  products  with  promise. 

Over  the  course  of  Select21,  less  dialogue  occurred  outside  the  Army  with  two  notable 
exceptions.  While  not  thoroughly  familiar  with  the  project,  the  MAPWG  is  aware  of  Select21 
and  has  encouraged  and  endorsed  the  research  on  non-cognitive  measures.  In  addition,  the 
ASVAB  review  panel,  mentioned  earlier  in  the  report,  had  access  to  Select21  findings  and 
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reports,  and  these  contributed  to  the  panel’s  emerging  recommendations  to  expand  the  use  of 
non-cognitive  measures  in  Service  selection  and  classification  programs. 


Additional  Observations 

Implementation  planning  should  profitably  begin  as  a  collaborative  effort  between 
researchers,  policymakers,  and  other  stakeholders  to  develop  shared  goals  and  support  within  the 
Army  for  achieving  these  goals.  Such  collaborative  planning  would  include  support  for  data 
collection  to  answer  critical  research  questions.  One  such  question  concerns  the  robustness  of 
“laboratory”  findings  and  their  transfer  to  high  stakes  operational  settings.  Policymakers  and 
stakeholders  may  not  support  a  study  in  which  the  new  measures  are  administered  in  conjunction 
with  the  ASVAB  because  it  complicates  operational  testing.  However,  policymakers  and 
stakeholders  might  be  more  open  to  research  in  which  new  recruits,  immediately  after  the 
decision  to  enter  service,  take  the  new  Select21  measures  for  purposes  of  verifying  the  Army’s 
assessment  of  their  likely  success  in  the  MOS  to  which  they  have  been  assigned.  This  use  of  the 
new  measures  might  create  a  type  of  high  stakes  motivation  (e.g.,  needs  on  the  part  of  test  takers 
to  demonstrate  that  they  had  made  good  decisions  for  their  future)  characteristic  of  individuals  in 
an  actual  operational  setting.  Planning  for  support  should  also  identify  the  players  in  DoD  and 
the  Services  who  may  have  a  part  in  implementation.  For  those  who  dispute  the  need  for 
implementation,  Army  researchers  need  to  take  steps  to  deal  with  their  objections  and  to  gain 
their  understanding  of  the  new  measures’  potential  for  enhancing  policy  and  operational 
procedures. 

Research  other  than  Select21  also  has  made  use  of  policymaker  and  SME  advisory 
panels.  Convening  and  retaining  such  panels  over  the  course  of  a  research  project  is  resource 
intensive.  However,  experience  has  shown  the  payoffs  of  this  investment  can  be  considerable. 
Panels  have  the  potential  of  overcoming  the  risks  created  by  the  periodic  transfer  of  military 
personnel.  That  is,  despite  rotation  of  individual  members,  the  group  itself  remains  stable  and 
can  help  to  motivate  new  members  and  to  incorporate  their  ideas  into  the  framework  of  the 
ongoing  effort.  With  the  correct  membership,  panels  might  help  to  overcome  gaps  in  the 
availability  of  resources  for  transitioning  research  products  into  practice.  Implementation  costs 
tend  to  fall  outside  the  missions  and  funding  of  research  organizations,  and  the  organizational 
element  that  benefits  from  research  innovations  (e.g.,  selection  and  classification)  is  not 
necessarily  the  organization  that  must  bear  the  costs  of  implementation  (e.g.,  USMEPCOM). 
Panels  initiated  early  in  the  research  process  and  composed  of  policymakers  and  stakeholders 
motivated  to  support  the  research  could  help  ensure  that  operational  organizations  impacted  by 
the  research  have  the  budget  and  other  resources  needed  for  its  implementation. 

Recommendations  for  Implementation 

As  noted  above,  implementation  is  a  crucial  component  in  moving  research  to  policy  and 
operational  practice.  Accordingly,  several  recommendations  are  offered  for  consideration. 
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•  Greater  attention  must  be  paid  to  implementation  planning.  Researchers  and 
policymakers  should  develop  an  implementation  plan  as  the  research  is  conducted. 
Consequently,  once  the  research  is  completed,  implementation  can  begin  in  a  timely  way. 

•  Funding  for  implementation  should  be  built  into  relevant  budgets.  Research  organizations 
should  be  involved  in  implementing  their  findings  consistent  with  their  missions  or 
mandates.  Thus,  funding  for  research  implementation  might  be  included  in  the  Research 
and  Development  budget.  On  the  other  hand,  funding  for  implementation,  more  often 
than  not,  should  be  included  in  the  budgets  of  either  the  relevant  policy  office  or  the 
operational  organization  affected  by  the  research.  Nevertheless,  no  matter  the  source, 
sufficient  funding  should  be  available  to  cover  the  cost  of  implementation. 

•  Enabling  legislation  and  new  regulations  should  be  prepared  as  part  of  implementation 
planning.  Implementation  strategies  take  time  and  resources;  having  authorizing 
directives  available  at  the  time  of  implementation  will  facilitate  the  process.  While 
researchers  can  contribute  to  development  of  authorizing  and  other  implementation 
directives,  having  such  directives  is  place  requires  planning  and  vigorous  promotion  by 
the  policymakers  with  interest  in  implementation. 

•  Researchers  should  be  sensitive  to  organizational  and  socio-political  factors  that  directly 
influence  implementation.  They  should  solicit  and  monitor  feedback  from  policymakers 
and  stakeholders  regarding  early  implementation  and  be  prepared  to  change  course  as 
needed.  This  includes  identifying  all  parties  who  may  have  a  part  in  the  implementation. 
For  those  who  have  problems  with  implementation,  it  is  necessary  to  overcome  their 
objections  and  explain  how  the  research  can  enhance  policy  and  operational  procedures. 
Such  feedback  can  guide  implementation  by  eliminating  any  barriers  encountered. 

•  Researchers  should  prepare  technical  reports  containing  sufficient  detail  to  inform 
implementation.  Technical  reports  should  be  written  for  policymakers  and  stakeholders, 
not  just  for  researchers.  Authors  should  include  and  discuss  results  that  will  be  highly 
relevant  to  implementation. 


Concluding  Comments 

Half-hearted  or  ill-advised  attempts  at  implementation  are  a  waste  of  time  and  money, 
and  may  frustrate  the  application  of  cutting-edge  research.  Research,  policy,  and  operational 
agendas  related  to  implementation  need  to  be  nurtured,  debated,  studied,  and  translated  into 
practical  advice  and  procedures  that  can  enhance  operational  systems.  We  are  optimistic  that 
Select21  can  and  will  advance  Army  selection  and  classification  as  new  principles  and 
procedures  are  illuminated  through  research  and  the  nexus  of  science,  policy,  and  practice. 
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